The Nesting-Depth of Disjunctive µ-Calculus for Tree Languages and the Limitedness Problem

In this paper we lift the result of Hashiguchi of decidability of the restricted star-height problem for words to the level of finite trees. Formally, we show that it is decidable, given a regular tree language L and a natural number k whether L can be described by a disjunctive μ-calculus formula with at most k nesting of fixpoints. We show the same result for disjunctive μ-formulas allowing substitution. The latter result is equivalent to deciding if the language is definable by a regular expression with nesting depth at most k of Kleene-stars. The proof, following the approach of Kirsten in the word case, goes by reduction to the decidability of the limitedness problem for nondeterministic nested distance desert automata over trees. We solve this problem in the more general framework of alternating tree automata.